Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Ollama, Palm, Claude-2, Cohere, Replicate, Llama2 CodeLlama (100+LLMs) [LiteLLM] #160

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add support for Ollama, Palm, Claude-2, Cohere, Replicate, Llama2 CodeLlama (100+LLMs) [LiteLLM] #160

wants to merge 4 commits into from

Conversation

ishaan-jaff
Copy link

This PR adds support for the above mentioned LLMs using LiteLLM https://github.com/BerriAI/litellm/
LiteLLM is a lightweight package to simplify LLM API calls - use any llm as a drop in replacement for gpt-3.5-turbo.

Example

from litellm import completion

## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["COHERE_API_KEY"] = "cohere key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion(model="command-nightly", messages)

# anthropic call
response = completion(model="claude-instant-1", messages=messages)

@ishaan-jaff
Copy link
Author

@loubnabnl @Muennighoff @infinitylogesh
can you take a look at this PR when possible ?

I believe LiteLLM makes it easier to benchmark LLMs - would love your feedback if not

@loubnabnl
Copy link
Collaborator

Thanks for the PR, what do you think @Muennighoff ?

Muennighoff
Muennighoff approved these changes Nov 15, 2023
View reviewed changes
Copy link
Contributor

@Muennighoff Muennighoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay with me if we add those two comments so people know that it's the same as via OAI

bigcode_eval/tasks/humanevalpack_openai.py Show resolved Hide resolved
bigcode_eval/tasks/humanevalpack_openai.py Show resolved Hide resolved
ishaan-jaff and others added 2 commits November 15, 2023 12:42
@ishaan-jaff @Muennighoff
Update bigcode_eval/tasks/humanevalpack_openai.py
2d74323 
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
@ishaan-jaff @Muennighoff
Update bigcode_eval/tasks/humanevalpack_openai.py
5188189 
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
@ishaan-jaff
Copy link
Author

@Muennighoff @loubnabnl updated based on feedback - thanks

loubnabnl
loubnabnl approved these changes Nov 16, 2023
View reviewed changes
Copy link
Collaborator

@loubnabnl loubnabnl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the integration! I left a small comment about the installation

setup.py
@@ -12,6 +12,7 @@
"matplotlib==3.5.2",
"numpy==1.21.6",
"openai==0.23.0",
"litellm==0.13.2",
Copy link
Collaborator

@loubnabnl loubnabnl Nov 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not the right place for the requirement as this is for DS1000 task that doesn't support external APIs, it seems we didn't have openai package properly added.

I would leave this list as it is and create a new variable
external_api_requirements=["litellm==0.13.2"] then add it to the extras_require dict here so people that want this evaluation install the package separately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Muennighoff Muennighoff Muennighoff approved these changes

@loubnabnl loubnabnl loubnabnl approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ishaan-jaff @loubnabnl @Muennighoff